Q-learning: Flexible learning about useful utilities

Dynamic treatment regimes are fast becoming an important part of medicine, with the corresponding change in emphasis from treatment of the disease to treatment of the individual patient. Because of the limited number of trials to evaluate personally tailored treatment sequences, inferring optimal treatment regimes from observational data has increased importance. Q-learning is a popular method for estimating the optimal treatment regime, originally in randomized trials but more recently also in observational data. Previous applications of Q-learning have largely been restricted to continuous utility end-points with linear relationships. This paper is the first both to extend the framework to discrete utilities and to implement the modelling of covariates from linear to more flexible modelling using the generalized additive model (GAM) framework. Simulated data results show that the GAM adapted Q-learning typically outperforms Q-learning with linear models and other frequently-used methods based on propensity scores in terms of coverage and bias/MSE. This represents a promising step towards a more fully general Q-learning approach to estimating optimal dynamic treatment regimes. This article is in technical report form, the final publication is available at http://www.springerlink.com/openurl.asp?genre=article&id=doi:10.1007/s12561-0139103-z .